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"^^ Abstract 

t~^ We study low rank matrix and tensor completion and propose novel algorithms with the best known 

T-H sample complexity guarantees for these problems. Our algorithms are active in that they interact with 

the sampling mechanism to obtain informative measurements. They are also sequential, processing the 

columns (sub-tensors) one at a time, and can easily be implemented in a streaming setting. For matrix 

completion, we show that one can exactly recover an n x n matrix of rank r using 0(r^nlog(r)) ob- 

^^ servations and for tensor completion, one can recover an order-T tensor using 0{r^^^^^'T^nlog{r)) 

. ' observations. We also establish a necessary condition for exact tensor completion from random observa- 

C^ tions. We complement our study with simulations that verify our theoretical guarantees and demonstrate 

T^ the scalability of our algorithms. 

^ 1 Introduction 

l> Computational resources and sensing technologies have failed to keep up with the rapid increases in the size 

\^ and complexity of modern scientific investigation. Insufficient processing power restricts analysts to using 

■^ heuristics and approximations that often fail to harness all of the information available in a dataset. At the 

■^ same time, costs associated with data acquisition often imply that datasets are highly unreliable, either due 

f^ to missing observations or low signal-to-noise ratios. Combined, both factors imply that modern data sets 

C^ have low information content, and what information is available cannot even be harnessed effectively. 

\ I As a concrete example, in biological sciences, a common practice is to use an initial high throughput 

J> assay to inform subsequent investigation, the latter often involving more robust - and more expensive - data 

collection. While high throughput technologies offer a low-cost solution to data acquition, these data sets 
are often incredibly high dimensional and very noisy. Meanwhile, acquisition costs prevent biologists from 
C^ using low-throughput technologies at scale, reflecting the tradeoff between reliable but costly data one one 

hand, and noisy but inexpensive data on the other. 

In the machine learning literature, the two main solutions approach the data acquisition problem from 
opposite sides. On one hand, statistical techniques for many high-dimensional problems are now near- 
optimal in an information theoretic sense, allowing scientists to effectively leverage high throughput tech- 
nologies. On the other hand, a number of researchers have shown that we can mitigate data acquisition 
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costs, particularly in settings where data is structured, by obtaining fewer measurements, demonstrating that 
low-throughput technologies can be intelligently deployed for large problems. Both strategies have been 
effective in theory and practice. 

Recently, Haupt et. al. ifTTI and Davenport et. al. |6| showed that adaptive data acquisition procedures, 
which use past measurements to inform subsequent ones, bring performance improvements over the best 
passive methods. Both results show that, for fixed sensing budget, one can recover a sparse vector much 
more accurately by adaptively focusing measurements onto components with signal. More broadly, a large 
body of work has shown that active learning often enjoys significantly better sample complexity bounds 
than passive methods |10|. While active learning is often quite statistically efficient, its role in leading to 
computational gains is largely unexplored. 

Motivated by the success of adaptive sensing in sparse vector recovery, we study the low rank matrix 
completion problem, a natural generalization. This problem involves recovering a low rank matrix from a 
subset of its entries and has been applied to recommender systems |19|, biological sciences 1 17 1, network 
tomography |7|, and several other domains. The numerous applications have spurred theoretical study of the 
problem, and it is now well-known that one can recover a low rank matrix using a vanishingly small fraction 
of randomly sampled entries ifTSll . 

Further generalizing, we also study low rank tensor completion. This problem is more relevant to 
biomedical imaging and video processing 1 15] but is far less known than matrix completion. To our knowl- 
edge, there are no known statistical guarantees for exact tensor completion. 

1.1 Contributions 

In this paper, we develop sequential algorithms with adaptive sampling schemes for matrix and tensor com- 
pletion. The algorithms process the matrix (tensor) in one pass in a streaming fashion, using the subspace 
detection test of Balzano et. al. 121 to select a few columns to measure in their entirety. The streaming nature 
of the algorithms lead to substantial benefits in both running time and memory overhead. 

In terms of sample complexity, we prove that our algorithm for matrix completion (Algorithm [T]i im- 
proves on the sample complexity bound of Recht i fTSJ , demonstrating the best known upper bound for ma- 
trix completion. Adaptive sampling is a instrumental in obtaining our bounds as they are lower than the 
necessary conditions for matrix completion from randomly sampled entries |5|. This establishes that our 
procedure outperforms all passive matrix completion algorithms. 

For the tensor completion problem, we analyze a non-trivial generalization of our matrix completion 
algorithm and estabhsh sample complexity bounds that scales with the sum of the tensor dimensions. This 
improves on the most natural generalization of Algorithm [T] which has sample complexity depending on 
the product of the tensor dimensions. We complement this upper bound with a lower bound for tensor 
completion under random sampling, showing that our adaptive strategy out-performs any passive algorithm. 
We show that this algorithm is also computationally fairly efficient. These are the first sample complexity 
upper and lower bounds for exact tensor completion. 

Broadly speaking, our work, along with 1.11. ^ demonstrates that adaptivity can further reduce data 
acquisition costs, even over the best passive schemes. At the same time, adaptive sampling can lead to 
computationally efficient algorithms that scale to large data sets. We hope this motivates subsequent study 
of adaptive data acquisition schemes, both in theory and practice. 



2 Related Work 

The matrix completion problem has received considerable attention in recent years. A series of papers 
Engl [141 |5]|T8l, culminating in Recht's elegent analysis of the nuclear norm minimization program, ad- 
dress the exact matrix completion problem through the framework of convex optimization, establishing that 
O(rmax{/xo, /x^}(ni +^2) log^(ft2)) randomly drawn samples are sufficient to exactly identify an ni x Ti2 
matrix with rank r. Here /ig and /ii are parameters characterizing the incoherence of the matrix, which we 
will define shortly. Candes and Tao |5| proved that under random sampling Vt{niiXQr\og{n2)) samples are 
necessary, showing that nuclear norm minimization is near-optimal. 

Tensor completion is less well known, despite being a natural generalization of matrix completion. The 
main challenge stems from the NP-hardness of computing most tensor decompositions, pushing researchers 
to study alternative structure-inducing norms in lieu of the nuclear norm [8,^20J. Both papers derive algo- 
rithms for tensor completion, but neither provide sample complexity bounds for the noiseless case, which 
we study here. Tomioka et. al. f2T| do derive bounds for the noisy setting, but to our knowledge, our results 
are the first such guarantees for exact tensor recovery. 

Our approch to the matrix completion problem involves adaptive data acquisition, and consequently 
our work is closely related to a number of papers focusing on estimating or localizing a sparse, possibly 
structured, vector corrupted by noise using adaptive measurement design IT] |6] (TT] [TJ] ■ In these problems, 
specifically, problems where the sparsity basis is known a priori, we have a reasonable understanding of how 
adaptive sampling can lead to performance improvements. As a low rank matrix is sparse in its unknown 
eigenbasis, the completion problem is coupled with learning this basis, which poses a new challenge for 
adaptive sampling procedures. 

Our algorithms, which learn the column space of the matrix by sequentially processing the columns, 
are also closely related to ideas employed for subspace detection - testing whether a vector lies in a known 
subspace - and subspace tracking - learning a time-evolving low dimensional subspace from vectors lying 
close to that subspace. Balzano et. al. {1 \ prove guarantees for subspace detection with known subspace and 
a partially observed vector, and we will leverage their results heavily in our analysis. Subspace tracking from 
partial information has been studied by both He et. al. lfT3l and Mateos and Giannakis [16|, who propose 
stochastic gradient-style algorithms, but little is known theoretically about this problem. 

3 Definitions and Preliminaries 

Before presenting our algorithms and theoretical guarantees, we clarify some notation and definitions. Let 
M — UYjV* e C"i ^ "^ be a rank r matrix of complex entries. We refer to the columns of i\/ as ci , . . . c„2 . 
Let M e £niy---xnT denote an order T tensor of complex entries. The CANDECOMP-PARAFAC 
decomposition factors M into a sum of rank one tensors: 



E 

fe=i 



(1) (2) (T) ,,s 



where o denotes the outer product. Our results focus on low rank tensors, where rank(M) is the smallest 
value of r that establishes this equality. Note that the vectors [a], Yk=i need not be orthogonal, nor even 
linearly independent G C"* , for fixed t. 

The mode-i subtensors of M, denoted M,- , i e [rit], are order T — 1 tensors obtained by fixing the ith 

(3) 

coordinate of the t-th mode. For example, if M is an order three tensor, then M^ are the frontal slices. 



We represent a d-dimensional subspace U C C" as a set of orthonormal basis vectors U = {ui\f^^ 
and in some cases as n x rf matrix whose columns are the basis vectors. It will be clear from context which 
interpretation we are using. Define the orthogonal projection onto U as: 

ruV = y u. 



Where (•, •) is the usual notion of inner product. 

For a set J7 C [n], c^ £ C'^' is the vector whose elements are Cj,i e J7 indexed lexicographically. 
Similarly the matrix Un G cl^^^lx'' has rows indexed by il lexicographically. Note that C/q is a |0| x d 
matrix with columns Uia where Ui G U, rather than a set of orthonormal basis vectors. 

These definitions extend to the tensor setting with slight modifications. We use the vec operation to 
unfold a tensor into a single vector with refold as the inverse operation. One readily verifies that {x, y) = 
vec(a;)-'"vec(y). For a subspace U C C^"*, we write it as a (H^-i) ^ d matrix whose columns are 
vec(ui), Ui E U. We can then define projections and subsampling as we did in the vector case. 

As in recent work on matrix completion |5__L8 1, we require a certain amount of incoherence between the 
subspaces associated with M (M) and the standard basis. We employ the usual definition of coherence: 

Definition 3.1. The coherence of an r-dimensional subspace U C C" is: 

m(C/)= "" max WVuejW^ (2) 

r l<j<n 

where Cj denotes the jth standard basis element. 

In previous analyses of matrix completion, the incoherence assumption is that both the row space and 
column space of the unknown matrix have coherences upper bounded by fiQ. Candes and Recht |4| demon- 
strated that some notion of incoherence is necessary to obtain non-trivial guarantees for matrix completion 
under random sampUng. Even under active sampling, it is impossible to identify a rank one matrix that is 
zero in all but one entry without observing the entire matrix. Requiring that the matrix subspaces have low 
coherence precludes these degenerate cases. 

Some fairly simple calculations, which we defer to the appendix, reveal some properties of incoherent 
subspaces, that will play a role in our analysis. 

Lemma 3.1. Let Ui C C"i , U2 C C"^ , ■ ■ -Ut C C"^ be subspaces of dimension at most d, let W\ C U\ 
have dimension d' . Define S = span{{Qj^iUl }f^i). Then: 

(a) m(W^i) < '-^^KUi). 

(b) Ai(§) < d^-^ uti m(c/.). 
4 Matrix Completion 

Our algorithm for matrix completion sequentially builds the column space of the unknown matrix M by 
selecting a few columns of the matrix to observe in their entirety. We maintain a candidate column space U 
and for each column c^, we test whether c^ e [/ or not, choosing to completely observe q if it does not lie 
in U. A key insight, which was observed by Balzano et. al. |2|, is that this test can be performed reliably 



Algorithm 1 Sequential Matrix Completion (M, to) 



1. Let[/ = 0. 

2. Randomly select fl C [ni] with \fl\ = m. 

3. For each column q of AI: 

(a) If||(/-Pj^Jc,o||i>0: 

1. Ci ^ Ci, Ui -It- ipp— ^T-jJ- 

ii. C/^ U\j{ui\. 

(b) Otherwise c, = U{U^Un)-^U'nC^n 

4. Return M with A'h = c,. 



by subsampling the coordinates of c^. The other key insight, is that active sampling can allow us to exactly 
identify a direction outside of U that belongs to the column space of M. 

The formal description of the sequential matrix completion algorithm is given in Algorithm [T] In more 
detail, at every iteration, we randomly sample a set of m coordinates of the column Ci and compute the 
projection of c^jj onto the orthogonal complement of U^. This projection serves to test whether c^ S [/ or 
not. If it is, then we already have enough information to recover it; if not, we observe the entire column and 
add it to our subspace U . 

Computationally, AlgorithmfTlis efficient in both time and space. In terms of running time, each iteration 
of the algorithm predominately operates with matrices of size \Vt\ x r and vectors of length Vt. The few 
iterations where we add a column to U, we must perform the Gram-Schmidt process, which is also efficient. 
In contrast, existing algorithms for matrix completion are iterative in nature and compute a singular value 
decomposition on each iteration, leading to considerable computational overhead |3|. 

The memory requirements of the algorithm are also minimal; indeed it is sublinear in ni x n2, the 
number of entries of the matrix. The algorithm streams the columns into memory, maintains the column 
space U and only needs to maintain the coefficients of each column. This amounts to at most [rii + 71,2) x r 
parameters, implying that the algorithm can scale to incredibly large datasets, as we show in Sectionl6] 

Our main result for matrix completion characterizes the performance of Algorithm [T] in terms of both 
sample complexity and running time. The only assumption is that the column space of M is incoherent. 

Theorem 4.1. Let M := UY.V* e C"!^"^ have rank r, and fix S > 0. Assume /i([/) < ^o- Setting 
m > 24r^^Q log(-j^), Algorithm\l\exactly recovers M with probability at least 1 — ArS + 5 while using at 
most: 

24n2rVo log(2r/5) + rm (3) 

observations. Algorithminruns in 0{nin2r + r^m) time. 

We make a few comments about the theorem, before providing a proof sketch. Recht fTSl recently 
guaranteed exact recovery for the nuclear norm minimization procedure as long as the number of observa- 
tions exceeds 32r(ni + 712) max{/xo; Mi}/3 log (2n2) where /3 controls the probability of failure and can be 
thought of as a constant and 1 1 UV* | |oo < Mi \/r/(nin2) with /ii as another coherence parameter. Without 
any additional assumptions, /ii can be as large as r^JJiQ. In this case, our bound matches his exactly except 
in logarithmic terms and constants, where our bound brings about substantial improvement. 



It is also worth noting that we make no assumptions other than the incoherence of the column space U. 
In particular, the row space V can be highly coherent without affecting the performance of our algorithm. 
All previous results require that V is also incoherent. 

We can also compare our results to known lower bounds. In the passive setting, under uniform sampling, 
Candes and Tao [5| showed that ^{iiQrn2log{ni / 6)) samples are necessary for completion of a rank r 
matrix whose subspaces have incoherence bounded by jiq. Our algorithm improves on this lower bound in 
the logarithmic factors, demonstrating that active sampling provides advantages over any passive algorithm. 
Unfortunately, just like previous results, Algorithm[T]deviates from the lower bound in its dependence on r 
and Ho, and it is an interesting open question to tighten this gap. 

In the active setting, a parameter counting argument reveals that a rank r matrix has r{ni + n2 — r) 
degrees of freedom. Observing the (?, j)th entry Alij of the matrix leads to a polynomial equation of the 
form J2k '^k{i)ck'Vk{j) ~ Mij. If m < r{ni + n2 — r) this system is underdetermined, has M as one 
solution, so it must have infinitely many solutions, showing that il{r{ni + 712)) observations are necessary 
for exact recovery, even under adaptive sampling. Thus, our algorithm enjoys sample complexity with 
optimal dependence on matrix dimensions. 

It seems that the additional dependence on r in our bound is not an artifact of our analysis, but rather 
unavoidable for our algorithm and possibly any algorithm that process columns sequentially. This factor 
arises because with no assumptions, LemmalTTTa) is tight, meaning that any individual column can have 
incoherence r/xo, even though the column space has incoherence /iQ. If each column was itself incoherent - 
a much stronger requirement - then our algorithm enjoys an improved sample complexity guarantee. 



Proof sketch of Theorem 4.1 The proof of correctness has two components: (1) an analysis of the test. 



11(7 — Vjj )cisi|P, and (2) verification that the remaining columns are recovered exactly. For the former, 
we verify that, with high probability, the test identifies columns with energy orthogonal to the subspace U. 
Balzano et. al. ^ analyzed this test statistic and we adapt Theorem 1 in that paper to the following lemma, 
characterizing the behavior of projections under random sampling: 

Lemma 4.2. Suppose that U C U and a column Cj G U but Cj ^ U. Let 5 <l/e. Ifm > 24r^/Zo log(2r/^) 
then with probability > 1 — 45, ||(/— P^ )cjn|p > 0. Ifcj G U then with probability 1, \\{I—Pjj )cjq\\'^ = 
0. 

The lemma, which we prove in the appendix, shows that the test statistic can identify the columns with 
additional information about U using few observations. Since Algorithm [T] fully samples these columns, a 
union bound verifies that at the end of the algorithm, U = U with high probability. 

For step (2), if Ci e U, then as long as |il| is large enough, the matrix C/j^C/o will be invertible, so Ci 
will be recovered exactly. The probability that U^iUn is invertible is actually subsumed in the probability of 
success in Lemma [421 ^^ invertibility is necessary to compute the projection. 

In total, we sample m observations from each column, and completely sample an additional r columns. 
Setting m = 24r^/iQ \og{2r /5), gives us the sample complexity bound. 

Computationally, the algorithm has three main parts: r inversions of the matrix UqUq (taking 0{mr^) 
time), 712 projection computations (taking 0{nin2r)), and orthonormalizing the basis via the Gram-Schmidt 
process (taking O(nir^)). D 



Algorithm 2 Sequential Tensor Completion (M, mx) 



1. LetW = 0. 

2. Randomly select fi C Y[t=i ["■*] ^^'-^ 1^1 ~ ''^t- 

3. For each mode-T subtensor M^^-* of M, i e [tit]: 

i. 0^ f-recurseon(Mj^^\rnT-i) 
ii. U, ^ " }t) ■ 
iii. U ^UUV^. 
(b) Otherwise MfVz^(Z^*Z^o)"'Z^nM^(^^ 

- (T) 

4. Return M with mode-T subtensors M; 



5 Tensor Completion 

With very few adjustments, AlgorithmfTlcan also be applied to the tensor completion problem. Let S denote 
span({Mj Yi^i) (the span of the mode-T subtensors). It is easy to see that S has dimension at most r, since 

it is also the span of {a), o a), o . . . o a). }k=i- We could build this subspace sequentially, using the 
same test statistic as before and completely observing each mode-T subtensor that did not completely lie in 
our current subspace. 

Unfortunately, applying the analysis of Theorem |4.1| reveals that the sample complexity of this algo- 
rithm is ?'(nt=i "^t) + "■T'^^^'^^^Vo log(2r/^), under the assumptions that the subspaces A^*' = 
span({aj }i=i) ^H have coherence bounded by /j,o- In contrast, a parameter counting argument reveals 
that there are only r J2t=i ^t degrees of freedom in the tensor, urging us to strive for better algorithms. In 
particular, we would like to significantly improve our dependence on ni. 

The weakness of the above algorithm is that it does not exploit additional structure in the mode-T sub- 
tensors, observing them as needed rather than attempting to complete them. The recursive version of that 
algorithm, which only fully observes individual fibers of the tensor and completes everything else, has sig- 
nificantly improved sample complexity, and is the algorithm we study in the remainder of this section. The 
pseudocode is provided in Algorithmic] 

Theorem 5.1. LefM = X^I^i OtLi^,- be a rank r order-T tensor with subspaces A^*' = span{{a, }^=i). 

Suppose that all of A^^' , . . .A^'^~^' have coherence bounded above by fiQ- Set nit = 24r^(*~^'/ig \og{2r/S) 

for each t. Then with probability > 1 — 5dTr^, Algorithm^exactly recovers M using no more than 



t=i 
observations. The running time of Algorithm^is: 

nnJ+r^^+M (5) 




Setting S = o(l/r^) we see that with probabiUty 1 — o(l) Algorithm0recovers M and uses 



t=i 



measurements, which is Hnear in the dimensions of the tensor, demonstrating a significant improvement 
over the straightforward appHcation of Algorithm [T] to the tensor case. The dependence on r and iiq seem 
undesirable, but as we will show, these dependencies are almost unavoidable. 



Before presenting our lower bound, we briefly compare Theorem 5.1 to existing results for tensor com- 



pletion. While the problem has not received much attention, there are some algorithms and some theoretical 
results that we can compare to. The most obvious idea is to unfold the matrix into a ni x Y[t=2 ^t ma- 
trix, which must have rank r, and apply any matrix completion algorithm. However, existing lower bounds 
show that the sample complexity of this approach will scale with Y[t=2 "^*' which is much worse than our 
guarantee. 

To our knowledge, the only known theoretical analysis of tensor completion is by Tomioka et. al. PH, 
who study the noisy version of the problem. They study an optimization involving the nuclear norms of the T 
different unfoldings of the tensor, but their results imply consistency in Frobenius norm only 51 1 r Y[t=2 "^t ) 
samples. Again this sample complexity has significantly worse dependence on the tensor dimensions than 
our algorithm. 

Theorem 5.2 (Passive Lower Bound). Fix I < m,r < mint "•« and /io > 1. Fix Q < S < 1/2 and suppose 
that we do not have the condition: 

-logfl--^)>^|^logg) (6) 

Then there exist infinitely many pairs of distinct rii x . . . x n^ order-T tensors M ^ M' of rank r with 
coherence parameter < /ip such that Vfi{M.) — Vn{^') with probability at least d. Each entry is observed 
independently with probability T = YFr' — ■ 

Theorem |5.2| implies that as long as the right hand side of Equation [6] is at most e < 1, and: 

m<niMrV^-Mog(|)(l-e/2) (7) 

then with probability at least 5 there are infinitely many matrices that agree on the observed entries. This 
gives a necessary condition on the number of samples required for tensor completion. 
We make two remarks about the lower bound: 

1 . The lower bound holds under the Bernoulli sampling model rather than the Uniform-at-random model. 
Candes and Tao fSl show how to translate results for the bernoulli model to the uniform model and 
since their proof works here as stated, we do not dive into these details. 

2. Equationr? shows that the factors of r^^^/ip ~^ are necessary, while our guarantee for Algorithm^ 
has factors of r^^ > Pf^ . As before, the extra factors of r arise from the fact that while each 
subspace is incoherent, each individual sub-tensor may not be; making the stronger assumption that 
each subtensor is also incoherent would improve the dependence on r. 
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Figure 1: Probability of success curves for Algorithm[T]and SVT. Success probability as a function of; Left: 
p, the fraction of samples per column, Center: np, total samples per column, and Right: np log^ n, expected 
samples per column for passive matrix completion. 
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Figure 2: Probability of success curves for Algorithm^ 



of: (a) p, the fraction of samples per subtensor and [(b) 



on order-3 tensors. Success probability as a function 
np, total samples per subtensor. 



6 Simulations 

We verify some of our theoretical results through several simulations. Our primary goal is to empirically 
demonstrate the sample complexity guarantees for Algorithm [T] and l2] We also study the scalability of our 
algorithms and compare to some existing methods for matrix completion. 

We verify Algorithm[T]s linear dependence on n in Figure[T] where we empirically compute the success 
probability of the algorithm for varying values of n and p = m/n, the fraction of entries observed per 
column. Here we study square matrices of fixed rank r = 5 with /i(f7) = 1. Figure 1(a) shows that 
AlgorithmfTlcan succeed with sampling a smaller and smaller fraction of entries as n increases, as we expect 
from Theorem |4.1 1 In Figure 1(b) we instead plot success probability against total number of observations 
per column. The fact that the curves coincide suggests that the samples per column, m, is constant with 
respect to n, which is precisely what Theorem 4.1 implies. Finally, in Figure 1(c) we rescale instead 



by n/log^n, which corresponds to the passive sample complexity bound ifTSll . Empirically, the fact that 
these curves do not line up demonstrates that Algorithm 111 requires fewer than log^ n samples per column, 
outperforming the passive bound. 
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84 
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0.07 


283 


6e- 11 



Table 1: Computational performance of Algorithm [T] on large low-rank matrices, d^ — r{2n 
degrees of freedom, so m/dr is the oversampling ratio, m/ri^ is fraction of entries. 



f) is the 



The second row of Figure [Tlplots the same probability of success curves for the Singular Value Thresh- 
olding (SVT) algorithm |3|. As is apparent from the plots, SVT does not enjoy a linear dependence on 
n; indeed Figure 1(f) confirms the logarithmic dependency that we expect for passive matrix completion. 



Comparing SVT to Algorithm [T] via these thresholds establishes that Algorithm [T] has empirically better 
performance. 

To confirm Algorithm fTts improvement in computational complexity over existing methods, we ran 
AlgorithmfTlon large-scale matrices, recording the running time and error in Tablefl] To contrast with SVT, 
we refer the reader to Table 5. 1 in |3 1. As an example, recovering a 10000 x 10000 matrix of rank 100 takes 
close to 2 hours with the SVT, while it takes less than 5 minutes with Algorithmfl] 

For Algorithm |2] we also confirm the linear scaling on dimension n with the probability of success 
curves in Figure|2] Here we fixed mt — m for all t,r = 5 and T = 3 and plot the probability of success as a 
function of p, the fraction of entries observed on order-2 tensors and m the total number of entries observed 
on each subtensor. As before, the fact that the curves line up in Figure 2(b) shows that Algorithml2]requires 
a constant number of observations per subtensor, confirming the linear scaling on tensor dimensions. 



7 Conclusions and Open Problems 

In this work, we demonstrate how algorithms that are both active and sequential can offer significant im- 
provements in time, space, and measurement overhead over passive algorithms for matrix and tensor com- 
pletion. These algorithms are incredibly appealing from a practical perspective, motivating further study of 
sequential active algorithms for machine learning. 

One short-coming of our work is that we do not study the noisy versions of matrix and tensor com- 
pletion. Robustifying our algorithms to noise turns out to be non-trivial; while one can analyze the test 
1 1(/ — Vfj )ca I P when the matrix is corrupted, the challenge involves controlling the deviation between the 
recovered column space U and the the true column space U. We intend to explore this direction further, in 
hopes of developing a more practical algorithm. 

Several interesting theoretical questions arise from our work: 

1 . Can we tighten the dependence on rank and incoherence parameter for these problems? 

2. Can one generalize the nuclear norm minimization program for matrix completion to the tensor com- 
pletion setting while providing theoretical guarantees on sample complexity? 



10 



3. Are there tighter lower bounds for tensor completion under random sampling that scale with the tensor 
order? 
We hope to pursue these directions in future work. 
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A Proof of Lemma 13.11 



For the first property, since VFi is a subspace of C/i, T^VFiGj —Vwi'Pui&j ^'^ WPwie-jW^. < W^Ui^jW^- The 
result now follows from the definition of incoherence. 

For the second property, we instead compute the incoherence of: 



S' = spanf|ori«(*)| 



which clearly contains S. Note that if {u\ } is an orthonormal basis for Ut (for each i), then the outer 
product of all combinations of these vectors is a basis for §'. We now compute: 



^l{W) 



< 



-iL=i^ max m'iot^e,M 

Ht^l dim([/t) fcie[ni],...,fcre[nT] 

nLi^^ nmaxV(»f)-efc,)^ 
nLdim(f/,)/i - ^' 

T 



Now, property (a) establishes that /i(S) < ^^/i(§') which is the desired result. 

B Proof of Theorem O 

Theproof of correctness begins by ensuring that for every column Cj, if Cj ^ [/ then \\{I — Vfj )cjfi\\'^ > 
with high probability. This is Lemma |4~2| which we prove here. We first state Theorem 1 from |,2J, which 
we then adapt to establish our Lemma: 



Theorem B.l. [2] Let U be a d-dimensional subspace ofC^ and v e C". Fix S > and let Q be a 
randomly sampled index set of size m > ^d^{U) log (^). Then with probability at least 1 — AS: 

^'-'^ \\v-Vuv\\l<\\vn-PunVn\\l (8) 

n 

And 

\\vn - VunVnWl < {I + a)-\\v - Vuv\\l (9) 



Where a 



\/^log(^). P ^ ^Mv)^)and^ = yMfilog(f). 
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Proof of Lemma |?!2] Noting that dfi{U)< r/ip, we can immediately apply Theorem B.l and are left to 
verify that the left hand side of Equation^is strictly positive. Since Cj ^ f/ we know that 1 1 {I~Vjj)ci | p > 0. 
Then: 



M^log(l/.)<J?^log(l/.) 



V ra 
Here we first used iJL{ci) < rfi{U) since c £ span([/). Finally, plugging in the definition of tti we arrive at 



a < 1/2. For 7 we have by Lemma 3.1a) 



^ - r^M'4)<-mM'4)<-i 



and for (1 + j3^) (as long as 5 < 1/e: 



(l + /3f < (1 + 2r/io log(l/<5) + 2^2rfio log{l/S)) 
< 6r/xo log(l/(5) 



so that: 



dA*(t/)^i^ < ^r;.(6r/.log(l/5)) < m/2 

1 — 7 2 

which completes the proof. D 

It is easy to see that if c^ G U then ||(/ — Vfj )cin\\'^ — deterministically and our algorithm does 
not further sample these columns. We must verify that these columns can be recovered exactly, and this 
amounts to checking that UqU[^ is invertible. Fortunately, this was established as a lemma in f2\, and in fact. 



the failure probability is subsumed by the probability in Theorem B.l 
Lemma B.2. Let 5 > Q and m > |r^o ^og{2r/5), Then: 

mUn)-^h<j^^ (10) 

with probability > 1 — (5, provided that 7 < 1. /n particular UqUq is invertible. 

Now we argue for correctness: there can be at most r columns for which ||(/ — Vq )cf2j|P > since 



rank(M) < ?-. For each of these columns, from Theorem B.l we know that with probability 1 — AS 
11(7 — Pjj )cni|p > 0. By a union bound, with probability > 1 — ArS all of these tests succeed, so the 
subspace U at the end of the algorithm is exactly the column space of M, namely U. All of these columns 
are recovered exactly, since we completely sample them. 

There are at most r + l index sets used throughout the algorithm, as we only update il when we add a col- 
umn to U. Except for the last index set, the probability that the corresponding matrices f/j^C/si are invertible 



is subsumed by the success probability of Theorem B.l In other words, the success of the projection test 



depends on the invertibility of these matrices, so the fact that we recovered the column space U implies that 



these matrices were invertible. The last such matrix is invertible except with probability 6 from Lemma B.2 
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If these matrices are invertible, then since Ci G U, we can write Ci = Uai and we have; 

So these columns are all recovered exactly. This step only adds a factor of S to the failure probability, leading 
to the final term in the failure probability of the theorem. 

For the running time, per column, the dominating computational costs involve the projection Vjj and 
the reconstruction procedure. The projection involves several matrix multiplications and the inversion of a 
rxr matrix, which need not be recomputed on every iteration. Ignoring the matrix inversion, this procedure 
takes at most 0{nir) per column for a total running time of 0{nin2r). At most r times, we must resample 
and recompute (UqUq)^^, which takes 0{r'^m), contributing a factor of 0(r^m) to the total running time. 
Finally, we run the Gram-Schmidt process once over the course of the algorithm, which takes 0{nir^) time. 
This last factor is dominated by nin2r. 



C Proof of Theorem 5.1 



We first focus on the recovery of the tensor in total, expressing this in terms of failure probabilities of the 
recursions. Then we apply an inductive argument to bound the failure probability of the entire algorithm. 
Finally we compute the total number of observations. For now, define tt to be the failure probability of 
recovering a T-order tensor 



dimension at most r. From Lemma 



By Lemma 3.1 the subspace s pann ed by the mode-T tensors as incoherence at most r fj. and 



4.2 



we see that with m > 24r^-^~^/j,g ~^ \og{2r/6) the projection test 
succeeds in identifying informative subtensors (those not in our current basis) with probability > 1 — 4(5. 
With a union bound over these r subtensors, the failure probability becomes < ArS + 6, not counting the 
probability that we fail in recovering these subtensors, which is ttt^i- 

For each mode T — 1 tensor that we have to recover, the subspace of interest has incoherence at most 
r'^~^fi^~'^ and with probability > 1 — ArS we correctly identify each informative subtensor as long as 
m > 24r^'^~^/i^"^~'* log{2r/S). Again the failure probability is < 4r(5 + S + rTT-2- 

To compute the total failure probability we proceed inductively, ti — since we completely observe 
any one-mode tensor (vector). The recurrence relation is: 

Tt = ArS + 6 + rn-i (11) 

which solves to: 

T-2 

TT = S + 4r^~^(5 + Y^ 5r*(5 < bSTr'^ (12) 

i = l 

We also compute the sample complexity inductively. Let rriT denote the number of samples needed to 
complete a T-order tensor Then mi — ni and: 

mt = rmt^i + 24ntr2*-Vo*"^ log{2r/S) (13) 

So that rriT is upper bounded as: 

T 

ruT = r^-ini+^r^-*24n,r2(*-i)^^(t-i)j^g(2r/J^ 



t=2 

< 24(f]n,)r2(^-i)^^(^-i)log(2r/J) 
t=i 
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The running time is computed in a similar way to the matrix case. Assume that the running time to 
complete an order t tensor is: 

T t 

Note that this is exactly the running time of Algorithm[T] so the base case is satisfied. 

Per order T — 1 subtensor, the projection and reconstructions take 0{r (nt=i ^*t))' which in total 

contributes a factor of 0{r ( Y[t=i ^t ) )■ At most r times, we must complete an order T — 1 subtensor, and 
invert the matrix U^Un- These two together take in total: 



O 



T-l T-1 



r(ll nt)+ y mtr^+'^ ^ ' 



rniT 



^(n-*)+E 

t=l t=2 

Finally the cost of the Gram-schmidt process is r^ Ilt^i "t which is dominated by the other costs. 
In total the running time is: 

(/ T \ T-l T \ 

\t=l I t=\ t=2 I 

/ / T \ T \ 



o . n«o+E 



mtr^+^-' 



\ \t=l / t=2 / 

since r < jit- Now plugging in that Wi = ©(r^*^*^^'), the terms in the second sum are each 0(r-'"+*+^) 
meaning that the sum is 0(r^^+^). This gives the computational result. 

D Proof of Theorem g^l 

We start by giving a proof in the matrix case, which is a slight variation of the proof by Candes and Tao ISj . 
Then we turn to the tensor case, where only small adjustments are needed to establish the result. We work 
in the Bernoulli model, noting that Candes' and Tao's arguments demonstrate how to adapt these results to 
the uniform-at-random sampling model. 

D.l Matrix Case 

In the matrix case, suppose that li = ^^ and I2 = ^^^ are both integers. Define the following blocks 

Ri, . . .Rr C [rii] and Ci, . . . C^ C [^2] as: 

R^ = {li{i-l) + l,li{i-l) + 2,...lii} 
C, = {I2ii-l) + l,l2{i-l) + 2,...l2i} 
Now consider the rii x n2 family of matrices defined by: 

r 

M = {^Ukvl\uk = [l,VMo]"ol^^,Wfe == IcJ (14) 

fc=i 
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Al is a family of block-diagonal matrices where the blocks have size li x ^2- Each block has constant rows 
whose entries may take arbitrary values in [1, ^/JIq]. For any M E Ai, the incoherence of the column space 
can be computed as: 

'^1 ™„,, ii-n . ii2 _ "1 _„„ _„„ (ulej)^ 



KU) - — maxllPc/e.ll^ 



max max 



r je[«i] r keM jelni] (uf.Uk)'^ 

< — max = fio 

r ke[r] ini/r) 

A similar straightforward calculation reveals that the row space is also incoherent with parameter /io- 

Unique identification of M is not possible unless we observe at least one entry from each row of each 
diagonal block. If we did not observe an entry in one such row, then we could vary that corresponding 
coordinate in the appropriate Uk and find infinitely many matrices M' E Ai that agree with our observations, 
have rank and incoherence at most r and /ig respectively. Thus, the probability of successful recovery is no 
larger than the probability of observing one entry of each row of each diagonal block. 

The probability that any single row of any block is unsampled is tti = (1 — p)'^ and the probability that 
all of the rows are sampled is (1 — Tri)"^. This quantity must upper bound the success probability I — S. 
Thus: 

-niTTi > ni log(l - TTi) > log(l -S)> -2(5 

or TTi < 26/ni as long as (5 < 1/2. Substituting tti = (1 — p)'^ we obtain: 

, /, X li /2(5\ ^0^, /2(5\ 
log(l -p) < ^ log — = ^ log — 

as a necessary condition for unique identification of M. 

Exponentiating both sides, writing p = — ^2_ ^nd the fact that 1 — e^^ > x — x^/2 gives us: 

m > nifiorlog (^-^j (1 - e/2) 
when fior/n2 log(fj) < e < 1. 

D.2 Tensor Case 

Fix T, the order of the tensor and suppose that ^i = ^^ is an integer. Moreover, suppose that k = ^^ is an 
integer for 1 < i < T. As in the matrix case, we will define a set of blocks, one for each mode of the tensor 
and the family of tensors 

Sf = {lt{^-l) + l,lt{^-l)+2,...Jt^}y^e[rlte[p] 

af^ = [1,\/Mo]"°1b(*) 



M = lJ20l=ia^ 



(t) 



a?^ -=1 „(*),! <t<T 



This is a family of block-diagonal tensors and just as before, straightforward calculations reveal that each 
subspace is incoherent with parameter /ig- Again, unique identification is not possible unless we observe 
at least one entry from each row of each diagonal block. The difference is that in the tensor case, there 
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are Yii^i ^i entries per row of each diagonal block so the probability that any single row is unsampled is 
TTi = (1 — pY^i*^ \ Again there are ni rows and any algorithm that succeeds with probability 1 — 5 must 
satisfy: 

-niTTi > ni log(l - TTi) > log(l -S)> -25 

Which implies tti < 25 /ni (assuming 5 < 1/2). When we substitute in the definition of tti we have: 

\og{l-p) < =. log — = ^ log - 

The same approximations as before yield the bound: 

m>mMrv^-Mog(|)(i-./2) 

as long as 4j^^ log(f^) < e < 1. 
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